Solving a Complex Prisoner's Dilemma with Self-Modifying Policies

Self-modifying policies (SMPs) trained by the success-story algorithm (SSA) have been successfully applied to various diicult reinforcement learning tasks (Schmidhuber et al. 1997a, 1997b). Here we present new results on an application where two cooperating/competing an-imats have to solve a complex version of the prisoner's dilemma. 1 Overview SMP/SSA. An animat's modiiable components that determine its behavior are called its policy. An algorithm that modiies the policy is called a learning algorithm. If the learning algorithm has modiiable components represented as part of the policy, then we speak of a self-modifying policy (SMP). SMPs can modify the way they modify themselves etc. They are of interest in situations where the initial learning algorithm itself can be improved by experience. How can we force some (stochas-tic) SMP to trigger better and better self-modiications? The success-story algorithm (SSA) addresses this question in a lifelong reinforcement learning context. During the learner's lifetime , SSA is occasionally called at times computed according to SMP itself. SSA uses backtrack-ing to undo those SMP-generated SMP-modiications that have not been empirically observed to trigger lifelong reward accelerations (measured up until the current SSA call | this evaluates the long-term eeects of SMP-modiications setting the stage for later SMP-modiications). SMP-modiications that survive SSA represent a lifelong success history. This scheme has been used to solve several diicult reinforcement learning tasks unsolvable by more traditional RL methods tasks (Schmidhuber et al., 1997a, 1997b). Outline. Basic principles of SMPs and SSA will be brieey reviewed in Section 2. Section 3 will describe an animat with a complex stochastic SMP, and Section 4 will present experiments with two such animats that need to learn to cooperate in a relatively complex environment to accelerate their reinforcement intake. In what follows we will review basic concepts presented in (Schmidhuber et al., 1997a). Each animat lives in an unknown environment E from time 0 to unknown time T. Life is one-way: even if it is decomposable into numerous consecutive \learning trials", time will never be reset. The animat has an internal state S and a self-modifying policy SMP. Both S and SMP are variable dynamic data structures innuencing probabilities of actions to be executed by the animat. Between time 0 and T, the animat repeats the following cycle over and over again (A denotes a set of possible actions): REPEAT: select and execute a 2 A with conditional probability P(a j SMP; S): 1 …